Kiva Loans - Find Similar Loans.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculate Similarity\n",
    "Given the diverse set of features here (about location, borrower, and the loan itself) let's just start with a really simple similarity metric.  We'll use Jaccard similarity, where each element of a user's set of loans will just get included in a set of loan elements.  Then we'll find a loan that's currently eligible with a large number of overlapping elements.\n",
    "\n",
    "We could eventually measure similarity across different metrics and weight more important ones more heavily, but to start let's just throw everything in the same set.\n",
    "\n",
    "The downside of this is that if someone consistently prefers the same country (for example) that's only indirectly accounted for in the small size of the set of elements (because that country would be counted only once, though it occurred many times.)  However, this will work as a proof of concept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import pprint\n",
    "import requests\n",
    "\n",
    "from country import country_to_continent\n",
    "from utils import eval_string\n",
    "\n",
    "path = '/Users/brianna/Dropbox/data_projects/loan_project/data/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get loan elements for each eligible loan\n",
    "Include the following features:\n",
    "- country\n",
    "- continent\n",
    "- sector\n",
    "- tags\n",
    "- themes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dictionary with loan ids and elements for each loan\n",
    "loan_elements = pickle.load( open( \"%sloan_elements.pickle\" % path, \"rb\" ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For a specific user, create a set of all of the elements of his/her previous loans.\n",
    "Let's look at my loan history for an example.  Because I lived in Uganda and I'm a big fan of woman-owned businesses, I tend to focus on East Africa and women."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "max_loans = 50\n",
    "NORMALIZE = None # None, 'sqrt' or 'random' to introduce random noise into results for more randomness  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_element(category, element):\n",
    "    if element not in category:\n",
    "        category[element] = 1\n",
    "    else:\n",
    "        category[element] += 1\n",
    "    \n",
    "    return category\n",
    "        \n",
    "def get_user_loan_elements(user):                                                                              \n",
    "    url = 'http://api.kivaws.org/v1/lenders/{user}/loans.json'.format(user=user)                               \n",
    "    response = requests.get(url)                                                                               \n",
    "    lender = eval(response.content.decode('utf8').replace(u'false', u'False').replace(u'true', u'True'))                          \n",
    "                                                                                                               \n",
    "    # To speed up computing time, if a user has a ton of loans only use the most recent.                    \n",
    "    if len(lender['loans']) > max_loans:                                                                              \n",
    "        lender['loans'] = lender['loans'][-max_loans:]                                                                \n",
    "                                                                                                               \n",
    "    # Make dictionaries of each of these important categories of the user's loans,\n",
    "    # where the key is the category (ex. \"Woman Owned Biz\") and the value is the number\n",
    "    # of times it's appeared in this user's loans\n",
    "    countries, continents, sectors, tags, themes = {}, {}, {}, {}, {}                       \n",
    "                                                                                                               \n",
    "    for loan in range(len(lender['loans'])):                                                                   \n",
    "        if 'country' in lender['loans'][loan]['location']:\n",
    "            country = lender['loans'][loan]['location']['country']\n",
    "            countries = add_element(countries, country)\n",
    "            continent = country_to_continent.get(country, 'Unknown')\n",
    "            continents = add_element(continents, continent)       \n",
    "                                                                                                               \n",
    "        if 'sector'in lender['loans'][loan]: \n",
    "            sector = lender['loans'][loan]['sector']\n",
    "            sectors = add_element(sectors, sector)                                                \n",
    "                                                                                                               \n",
    "        if 'tags' in lender['loans'][loan]:                                                                    \n",
    "            loan_tags = [k['name'].strip('#') for k in lender['loans'][loan]['tags']]  \n",
    "            for t in loan_tags:\n",
    "                tags = add_element(tags, t)                                                                             \n",
    "                                                                                                               \n",
    "        if 'themes' in lender['loans'][loan]:                                                                  \n",
    "            loan_themes = lender['loans'][loan]['themes'] \n",
    "            for th in loan_themes:\n",
    "                themes = add_element(themes, th)                                                                       \n",
    "\n",
    "        user_loan_elements = {'user_countries': countries,                                                     \n",
    "                              'user_continents': continents,                                                   \n",
    "                              'user_sectors': sectors,                                                         \n",
    "                              'user_tags': tags,                                                               \n",
    "                              'user_themes': themes}                                                           \n",
    "                                                                                                               \n",
    "    return user_loan_elements   \n",
    "\n",
    "def get_user_loan_elements_and_counts(user_loan_elements):                                         \n",
    "    # Combine elements from each category into a single set to calculate similarity         \n",
    "    user_loan_elements_set = {}\n",
    "    for category in user_loan_elements: \n",
    "        for element in user_loan_elements[category]:\n",
    "            user_loan_elements_set.update(user_loan_elements[category])\n",
    "    return user_loan_elements_set\n",
    "\n",
    "def get_user_loan_elements_categories_only(user_loan_elements):                                         \n",
    "    # Combine elements from each category into a single set to calculate similarity         \n",
    "    user_loan_elements_set = set()                                                          \n",
    "    for category in user_loan_elements:                                                           \n",
    "        user_loan_elements_set.update(user_loan_elements[category])                               \n",
    "    return user_loan_elements_set                                                           "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "user = 'brianna9306'\n",
    "user_loan_elements = get_user_loan_elements(user)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'user_continents': {'Africa': 18},\n",
       " 'user_countries': {'Kenya': 1, 'Uganda': 17},\n",
       " 'user_sectors': {'Agriculture': 2,\n",
       "  'Clothing': 2,\n",
       "  'Food': 7,\n",
       "  'Housing': 1,\n",
       "  'Retail': 6},\n",
       " 'user_tags': {'Animals': 1,\n",
       "  'Eco-friendly': 1,\n",
       "  'Fabrics': 3,\n",
       "  'Health and Sanitation': 1,\n",
       "  'Interesting Photo': 1,\n",
       "  'Job Creator': 1,\n",
       "  'Parent': 9,\n",
       "  'Repeat Borrower': 1,\n",
       "  'Schooling': 1,\n",
       "  'Single Parent': 1,\n",
       "  'Technology': 1,\n",
       "  'Unique': 1,\n",
       "  'Vegan': 2,\n",
       "  'Woman Owned Biz': 9,\n",
       "  'user_favorite': 10},\n",
       " 'user_themes': {'Green': 1,\n",
       "  'Growing Businesses': 1,\n",
       "  'Job Creation': 1,\n",
       "  'Rural Exclusion': 9,\n",
       "  'Social Enterprise': 1,\n",
       "  'Vulnerable Groups': 4,\n",
       "  'Youth': 4}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_loan_elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Fabrics', 'Africa', 'Job Creator', 'Repeat Borrower', 'Housing', 'Uganda', 'Vegan', 'Single Parent', 'Agriculture', 'Parent', 'Vulnerable Groups', 'Eco-friendly', 'Animals', 'Health and Sanitation', 'Growing Businesses', 'Rural Exclusion', 'Woman Owned Biz', 'Unique', 'Kenya', 'Schooling', 'Retail', 'Social Enterprise', 'Food', 'Green', 'Job Creation', 'user_favorite', 'Technology', 'Youth', 'Interesting Photo', 'Clothing'}\n",
      "\n",
      "\n",
      "{'Fabrics': 3, 'Africa': 18, 'Job Creator': 1, 'Housing': 1, 'Uganda': 17, 'Vulnerable Groups': 4, 'Single Parent': 1, 'Agriculture': 2, 'Vegan': 2, 'Eco-friendly': 1, 'Repeat Borrower': 1, 'Animals': 1, 'Growing Businesses': 1, 'Social Enterprise': 1, 'Rural Exclusion': 9, 'Retail': 6, 'Woman Owned Biz': 9, 'Unique': 1, 'Health and Sanitation': 1, 'Kenya': 1, 'Schooling': 1, 'Food': 7, 'Green': 1, 'Job Creation': 1, 'user_favorite': 10, 'Technology': 1, 'Youth': 4, 'Interesting Photo': 1, 'Clothing': 2, 'Parent': 9}\n"
     ]
    }
   ],
   "source": [
    "print(get_user_loan_elements_categories_only(user_loan_elements))\n",
    "print('\\n')\n",
    "print(get_user_loan_elements_and_counts(user_loan_elements))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Similarity Measures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Jaccard Distance\n",
    "The simplest way to compute similarity, the number of shared elements divided by the total number of elements.\n",
    "   - <b>Strengths</b>: Simple to understand, the fit is imperfect but it injects an element of randomness that might be beneficial\n",
    "   - <b>Weaknesses</b>: Doesn't take into account the nature of the category (ex. treats country the same as sector) or the number of times each category is is associated with the loans (ex. If I take loans 30 loans from Senegal and one from Cambodia, it counts Senegal and Cambodia equally)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def jaccard_distance(x, user_loan_elements):                                          \n",
    "    user_loan_elements_set = get_user_loan_elements_categories_only(user_loan_elements)           \n",
    "    intersection = len(set.intersection(x, user_loan_elements_set))                   \n",
    "    union = len(set.union(x, user_loan_elements_set))                                 \n",
    "    if union > 0:                                                                     \n",
    "        return intersection/float(union)                                              \n",
    "    else:                                                                             \n",
    "        return 0      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dot Product Similarity\n",
    "Compute similarity taking into account the number of instances of each element (ie. if someone gives 10 \"Agriculture\" sector loans and 2 \"Clothing\" sector loans, their similarity scores will be more heavily weighted toward \"Agriculture\".)\n",
    "\n",
    "   - <b>Strengths</b>: If a user has a strong preference in one category, that category will carry more weight, resulting in more loans that are similar in that category\n",
    "   - <b>Weaknesses</b>: Less intuitive measure, tailored specifically to the eccentricities of the Kiva loan dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_max_instance(user_loan_elements, category):                                                      \n",
    "    # Get the maximum number of instances of a particular category (country, continent, sector)          \n",
    "    instances = {v: k for k,v in user_loan_elements[category].items()}                                   \n",
    "    if NORMALIZE == 'sqrt':                                                                              \n",
    "        return math.sqrt(max(instances))                                                                 \n",
    "    elif NORMALIZE == None or NORMALIZE == 'random':                                                     \n",
    "        return max(instances)                                                                            \n",
    "    else:                                                                                                \n",
    "        raise ValueError('NORMALIZE must be None, sqrt, or random')                                      \n",
    "                                                                                                         \n",
    "def get_sum_of_instances(user_loan_elements, category):                                                  \n",
    "    # Get the sum of number of instances of a particular category (tags, themes)                         \n",
    "    # Since the maximum number of tags and themes for any given loan is about 3,                         \n",
    "    # only take the top 3 instances                                                                      \n",
    "    total_instances = []                                                                                 \n",
    "    for k in user_loan_elements[category]:                                                               \n",
    "        if NORMALIZE == 'sqrt':                                                                          \n",
    "            total_instances.append(math.sqrt(user_loan_elements[category][k]))                           \n",
    "        elif NORMALIZE == None or NORMALIZE == 'random':                                                 \n",
    "            total_instances.append(user_loan_elements[category][k])                                      \n",
    "        else:                                                                                            \n",
    "            raise ValueError('NORMALIZE must be None, sqrt, or random')                                  \n",
    "    total_instances.sort()                                                                               \n",
    "    return sum(total_instances[-3:])                                                                     \n",
    "                                                                                                         \n",
    "\n",
    "def dot_product_plus_random_noise(user_loan_elements_set, k):                                         \n",
    "    # Introduce an element of randomness so the user doesn't see 15 loans that are basically          \n",
    "    # identical and the closest fit.  Pick a random number from zero to the true number and           \n",
    "    # subtract it from the real number.                                                               \n",
    "    num_shared = user_loan_elements_set.get(k, 0)                                                     \n",
    "    noise = random.uniform(0, num_shared)                                                             \n",
    "    return num_shared - noise                                                                         \n",
    "                                                                                                      \n",
    "def dot_product_sqrt(user_loan_elements_set, k):                                                      \n",
    "    # Use the sqrt of the dp instead of the dp itself to give smaller numbers a fighting chance       \n",
    "    num_shared = user_loan_elements_set.get(k, 0)                                                     \n",
    "    return math.sqrt(num_shared)                                                                      \n",
    "                                                                                                      \n",
    "def max_dp_similarity(user_loan_elements):                                                            \n",
    "    # The maximum \"similarity score\" would be:                                                        \n",
    "    #    - a match with the most common country, continent, and sector                                \n",
    "    #    - all tags and themes shared                                                                 \n",
    "    dp_max = 0                                                                                        \n",
    "    for category in ['user_countries', 'user_continents', 'user_sectors']:                            \n",
    "        dp_max += get_max_instance(user_loan_elements, category)                                      \n",
    "    for category in ['user_tags', 'user_themes']:                                                     \n",
    "        dp_max += get_sum_of_instances(user_loan_elements, category)                                  \n",
    "    return dp_max                                                                                     \n",
    "    \n",
    "def dp_similarity(candidate_loan_elements, user_loan_elements):\n",
    "    user_loan_elements_set = get_user_loan_elements_and_counts(user_loan_elements)\n",
    "    candidate_loan_elements = list(candidate_loan_elements)\n",
    "\n",
    "    # To get the best possible fits, just take the dot product of each loan's elements with\n",
    "    # the user's loan elements.\n",
    "    if NORMALIZE == 'random':\n",
    "        # To inject some randomness in choices, inject a bit of random noise\n",
    "        dot_product = sum([dot_product_plus_random_noise(user_loan_elements_set, k) for k in candidate_loan_elements])\n",
    "    elif NORMALIZE == 'sqrt':\n",
    "        dot_product = sum([dot_product_sqrt(user_loan_elements_set, k) for k in candidate_loan_elements])\n",
    "    elif NORMALIZE == None:\n",
    "        dot_product = sum([user_loan_elements_set.get(k, 0) for k in candidate_loan_elements])\n",
    "    else:\n",
    "        raise ValueError('NORMALIZE must be None, sqrt, or random')\n",
    "\n",
    "    # Normalize by the maximum dot product of the most ideal loan (which is not the same as sharing\n",
    "    # every element since a loan can in theory share all tags and themes but can only have one\n",
    "    # country, continent, and sector\n",
    "    if dot_product > max_dp_similarity(user_loan_elements):\n",
    "        return 1.0\n",
    "    else:\n",
    "        return dot_product * 1.0 / max_dp_similarity(user_loan_elements)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Find most and least similar loans (Ja"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_best_and_worst_loans(user, similarity_metric):\n",
    "    \n",
    "    print('Loan profile for {}\\n'.format(user))\n",
    "    user_loan_elements = get_user_loan_elements(user)\n",
    "    for element in user_loan_elements:\n",
    "        print(element)\n",
    "        for sub_element in user_loan_elements[element]:\n",
    "            print('\\t{}: {}'.format(sub_element, user_loan_elements[element][sub_element]))\n",
    "        \n",
    "    # Find which of the currently active loans has the highest overlap with the user_loan_elements\n",
    "    # User either Jaccard or Dot Product metric to calculate similarity\n",
    "    if similarity_metric == 'jaccard':\n",
    "        similarity_scores = {k: jaccard_distance(v['elements'], user_loan_elements) for k, v in loan_elements.items()}\n",
    "    elif similarity_metric == 'dot_product':\n",
    "        similarity_scores = {k: dp_similarity(v['elements'], user_loan_elements) for k, v in loan_elements.items()}\n",
    "    else:\n",
    "        print(\"Similarity metric should be either 'jaccard' or 'dot_product'\")\n",
    "        return\n",
    "\n",
    "    # Turn the dict into a list of tuples sorted by similarity score\n",
    "    sorted_similarity_scores = sorted(similarity_scores.items(), key=lambda tup: tup[1], reverse=True)\n",
    "    \n",
    "    print('\\n\\n***TOP FIVE SIMILAR LOANS for {}, using {} similarity metric***'.format(user, similarity_metric))\n",
    "    for sim in sorted_similarity_scores[:5]:\n",
    "        loan_id, similarity_score = sim[0], sim[1]\n",
    "        print('\\nLoan: {}, similarity: {}'.format(loan_id, similarity_score))\n",
    "        print(loan_elements[loan_id]['elements'])\n",
    "\n",
    "\n",
    "    print('\\n\\n***TOP FIVE WORST LOANS for {}, using {} similarity metric***'.format(user, similarity_metric))\n",
    "    sorted_similarity_scores.reverse()\n",
    "    for sim in sorted_similarity_scores[:5]:\n",
    "        loan_id, similarity_score = sim[0], sim[1]\n",
    "        print('\\nLoan: {}, similarity: {}'.format(loan_id, similarity_score))\n",
    "        print(loan_elements[loan_id]['elements'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loan profile for brianna9306\n",
      "\n",
      "user_countries\n",
      "\tKenya: 1\n",
      "\tUganda: 17\n",
      "user_tags\n",
      "\tFabrics: 3\n",
      "\tSchooling: 1\n",
      "\tJob Creator: 1\n",
      "\tUnique: 1\n",
      "\tVegan: 2\n",
      "\tRepeat Borrower: 1\n",
      "\tWoman Owned Biz: 9\n",
      "\tAnimals: 1\n",
      "\tSingle Parent: 1\n",
      "\tParent: 9\n",
      "\tEco-friendly: 1\n",
      "\tuser_favorite: 10\n",
      "\tTechnology: 1\n",
      "\tInteresting Photo: 1\n",
      "\tHealth and Sanitation: 1\n",
      "user_sectors\n",
      "\tRetail: 6\n",
      "\tHousing: 1\n",
      "\tAgriculture: 2\n",
      "\tFood: 7\n",
      "\tClothing: 2\n",
      "user_continents\n",
      "\tAfrica: 18\n",
      "user_themes\n",
      "\tSocial Enterprise: 1\n",
      "\tGrowing Businesses: 1\n",
      "\tRural Exclusion: 9\n",
      "\tGreen: 1\n",
      "\tJob Creation: 1\n",
      "\tYouth: 4\n",
      "\tVulnerable Groups: 4\n",
      "\n",
      "\n",
      "***TOP FIVE SIMILAR LOANS for brianna9306, using jaccard similarity metric***\n",
      "\n",
      "Loan: 1315583, similarity: 0.3\n",
      "{'Africa', 'Repeat Borrower', 'Woman Owned Biz', 'Kenya', 'Retail', 'Schooling', 'Single Parent', 'Parent', 'user_favorite'}\n",
      "\n",
      "Loan: 1313614, similarity: 0.2903225806451613\n",
      "{'Rural Exclusion', 'Africa', 'Repeat Borrower', 'Woman Owned Biz', 'Kenya', 'Agriculture', 'Parent', 'user_favorite', 'Biz Durable Asset', 'Animals'}\n",
      "\n",
      "Loan: 1311270, similarity: 0.26666666666666666\n",
      "{'Africa', 'Repeat Borrower', 'Woman Owned Biz', 'Vegan', 'Kenya', 'Retail', 'Single Parent', 'Parent'}\n",
      "\n",
      "Loan: 1316269, similarity: 0.25\n",
      "{'Africa', 'Repeat Borrower', 'Kenya', 'Retail', 'Schooling', 'Earth Day Campaign', 'Parent', 'Green', 'user_favorite', 'Biz Durable Asset'}\n",
      "\n",
      "Loan: 1310375, similarity: 0.25\n",
      "{'Rural Exclusion', 'Africa', 'Repeat Borrower', 'Woman Owned Biz', 'Vegan', 'Mali', 'Parent', 'Food', 'Underfunded Areas', 'user_favorite'}\n",
      "\n",
      "\n",
      "***TOP FIVE WORST LOANS for brianna9306, using jaccard similarity metric***\n",
      "\n",
      "Loan: 1310713, similarity: 0.0\n",
      "{'Personal Use', 'Tajikistan', 'Asia'}\n",
      "\n",
      "Loan: 1302510, similarity: 0.0\n",
      "{'Pakistan', 'Transportation', 'Asia', 'Repair Renew Replace', 'Underfunded Areas'}\n",
      "\n",
      "Loan: 1302483, similarity: 0.0\n",
      "{'Pakistan', 'Asia', 'Transportation', 'Elderly', 'Repair Renew Replace', 'Underfunded Areas'}\n",
      "\n",
      "Loan: 1302477, similarity: 0.0\n",
      "{'Pakistan', 'Transportation', 'Asia', 'Repair Renew Replace', 'Underfunded Areas'}\n",
      "\n",
      "Loan: 1315082, similarity: 0.0\n",
      "{'Asia', 'Philippines', 'Transportation'}\n"
     ]
    }
   ],
   "source": [
    "# Check the best loans for me using the Jaccard similarity metric\n",
    "print_best_and_worst_loans('brianna9306', 'jaccard')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Four of my top five \"best\" loan matches are in Kenya, where I have given one loan.  However, the majority of the loans I've given (17 out of 18) were in Uganda.  The Jaccard metric takes into account whether an instance has ever happened, but not the number of times it has happened.\n",
    "\n",
    "The five \"worst\" loans are definitely subjectively different than loans I would normally be interested in.  I've never given (or been interested in) loans for transportation in Asia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loan profile for marylynn4377\n",
      "\n",
      "user_countries\n",
      "\tPhilippines: 20\n",
      "user_tags\n",
      "\tRepeat Borrower: 1\n",
      "\tJob Creator: 1\n",
      "\tParent: 2\n",
      "\tWoman Owned Biz: 6\n",
      "\tuser_favorite: 1\n",
      "\tElderly: 2\n",
      "\tRepair Renew Replace: 1\n",
      "\tAnimals: 8\n",
      "user_sectors\n",
      "\tHousing: 2\n",
      "\tAgriculture: 10\n",
      "\tFood: 7\n",
      "\tClothing: 1\n",
      "user_continents\n",
      "\tAsia: 20\n",
      "user_themes\n",
      "\n",
      "\n",
      "***TOP FIVE SIMILAR LOANS for marylynn4377, using jaccard similarity metric***\n",
      "\n",
      "Loan: 1310657, similarity: 0.5714285714285714\n",
      "{'Asia', 'Woman Owned Biz', 'Elderly', 'Philippines', 'Agriculture', 'Parent', 'user_favorite', 'Animals'}\n",
      "\n",
      "Loan: 1310753, similarity: 0.5\n",
      "{'Asia', 'Repeat Borrower', 'Woman Owned Biz', 'Philippines', 'Agriculture', 'Parent', 'Animals'}\n",
      "\n",
      "Loan: 1311963, similarity: 0.5\n",
      "{'Asia', 'Repeat Borrower', 'Woman Owned Biz', 'Elderly', 'Philippines', 'Agriculture', 'Animals'}\n",
      "\n",
      "Loan: 1309535, similarity: 0.5\n",
      "{'Asia', 'Repeat Borrower', 'Woman Owned Biz', 'Elderly', 'Philippines', 'Agriculture', 'Animals'}\n",
      "\n",
      "Loan: 1309484, similarity: 0.5\n",
      "{'Asia', 'Repeat Borrower', 'Woman Owned Biz', 'Philippines', 'Agriculture', 'Parent', 'Animals'}\n",
      "\n",
      "\n",
      "***TOP FIVE WORST LOANS for marylynn4377, using jaccard similarity metric***\n",
      "\n",
      "Loan: 1316276, similarity: 0.0\n",
      "{'Services', 'Palestine', 'Arab Youth', 'Unknown', 'Start-Up', 'Youth'}\n",
      "\n",
      "Loan: 1318880, similarity: 0.0\n",
      "{'Retail', 'Africa', 'Mozambique'}\n",
      "\n",
      "Loan: 1307165, similarity: 0.0\n",
      "{'Retail', 'North_America', 'United States'}\n",
      "\n",
      "Loan: 1319446, similarity: 0.0\n",
      "{'Retail', 'Africa', 'Mozambique'}\n",
      "\n",
      "Loan: 1308258, similarity: 0.0\n",
      "{'Rural Exclusion', 'Underfunded Areas', 'Innovative Loans', 'Myanmar (Burma)', 'Unknown', 'Personal Use', 'Youth'}\n"
     ]
    }
   ],
   "source": [
    "# Let's try someone with different preferences to see how it fits.\n",
    "print_best_and_worst_loans('marylynn4377', 'jaccard')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice, the top five \"best\" loans all involve women-owned businesses and animals in the Phillipines.  If anything, this might be recommending loans that are all too similar.  It will be good to introduce some noise here to add some diversity to the offerings.\n",
    "\n",
    "The five \"worst\" loans are unrelated to the Phillipines, animals, or women-owned businesses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Try Dot Product metric instead\n",
    "A dot product similarity metric will take into account the number of instances of a loan element. For example, if I've given 17 loans in Uganda and 1 in Kenya, Uganda will be weighted 17 times higher than Kenya."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loan profile for brianna9306\n",
      "\n",
      "user_countries\n",
      "\tKenya: 1\n",
      "\tUganda: 17\n",
      "user_tags\n",
      "\tFabrics: 3\n",
      "\tSchooling: 1\n",
      "\tJob Creator: 1\n",
      "\tUnique: 1\n",
      "\tVegan: 2\n",
      "\tRepeat Borrower: 1\n",
      "\tWoman Owned Biz: 9\n",
      "\tAnimals: 1\n",
      "\tSingle Parent: 1\n",
      "\tParent: 9\n",
      "\tEco-friendly: 1\n",
      "\tuser_favorite: 10\n",
      "\tTechnology: 1\n",
      "\tInteresting Photo: 1\n",
      "\tHealth and Sanitation: 1\n",
      "user_sectors\n",
      "\tRetail: 6\n",
      "\tHousing: 1\n",
      "\tAgriculture: 2\n",
      "\tFood: 7\n",
      "\tClothing: 2\n",
      "user_continents\n",
      "\tAfrica: 18\n",
      "user_themes\n",
      "\tSocial Enterprise: 1\n",
      "\tGrowing Businesses: 1\n",
      "\tRural Exclusion: 9\n",
      "\tGreen: 1\n",
      "\tJob Creation: 1\n",
      "\tYouth: 4\n",
      "\tVulnerable Groups: 4\n",
      "\n",
      "\n",
      "***TOP FIVE SIMILAR LOANS for brianna9306, using dot_product similarity metric***\n",
      "\n",
      "Loan: 1300114, similarity: 0.8045977011494253\n",
      "{'Rural Exclusion', 'Africa', 'Woman Owned Biz', 'Uganda', 'Schooling', 'Parent', 'Food'}\n",
      "\n",
      "Loan: 1310375, similarity: 0.7471264367816092\n",
      "{'Rural Exclusion', 'Africa', 'Repeat Borrower', 'Woman Owned Biz', 'Vegan', 'Mali', 'Parent', 'Food', 'Underfunded Areas', 'user_favorite'}\n",
      "\n",
      "Loan: 1301754, similarity: 0.7126436781609196\n",
      "{'Africa', 'Repeat Borrower', 'Uganda', 'Parent', 'Food', 'user_favorite'}\n",
      "\n",
      "Loan: 1312653, similarity: 0.7011494252873564\n",
      "{'Rural Exclusion', 'Africa', 'Repeat Borrower', 'Uganda', 'Retail', 'Schooling', 'Parent'}\n",
      "\n",
      "Loan: 1298809, similarity: 0.7011494252873564\n",
      "{'Africa', 'Rural Exclusion', 'Repeat Borrower', 'Uganda', 'Retail', 'user_favorite'}\n",
      "\n",
      "\n",
      "***TOP FIVE WORST LOANS for brianna9306, using dot_product similarity metric***\n",
      "\n",
      "Loan: 1310713, similarity: 0.0\n",
      "{'Personal Use', 'Tajikistan', 'Asia'}\n",
      "\n",
      "Loan: 1302510, similarity: 0.0\n",
      "{'Pakistan', 'Transportation', 'Asia', 'Repair Renew Replace', 'Underfunded Areas'}\n",
      "\n",
      "Loan: 1302483, similarity: 0.0\n",
      "{'Pakistan', 'Asia', 'Transportation', 'Elderly', 'Repair Renew Replace', 'Underfunded Areas'}\n",
      "\n",
      "Loan: 1302477, similarity: 0.0\n",
      "{'Pakistan', 'Transportation', 'Asia', 'Repair Renew Replace', 'Underfunded Areas'}\n",
      "\n",
      "Loan: 1315082, similarity: 0.0\n",
      "{'Asia', 'Philippines', 'Transportation'}\n"
     ]
    }
   ],
   "source": [
    "print_best_and_worst_loans('brianna9306', 'dot_product')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Four of the top five \"similar\" loans are rural and/or women-owned businesses in Uganda, nice!  \n",
    "\n",
    "And conversely, my five least good fits are in places where I've never given a loan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loan profile for marylynn4377\n",
      "\n",
      "user_countries\n",
      "\tPhilippines: 20\n",
      "user_tags\n",
      "\tRepeat Borrower: 1\n",
      "\tJob Creator: 1\n",
      "\tParent: 2\n",
      "\tWoman Owned Biz: 6\n",
      "\tuser_favorite: 1\n",
      "\tElderly: 2\n",
      "\tRepair Renew Replace: 1\n",
      "\tAnimals: 8\n",
      "user_sectors\n",
      "\tHousing: 2\n",
      "\tAgriculture: 10\n",
      "\tFood: 7\n",
      "\tClothing: 1\n",
      "user_continents\n",
      "\tAsia: 20\n",
      "user_themes\n",
      "\n",
      "\n",
      "***TOP FIVE SIMILAR LOANS for marylynn4377, using dot_product similarity metric***\n",
      "\n",
      "Loan: 1310753, similarity: 1.0\n",
      "{'Asia', 'Repeat Borrower', 'Woman Owned Biz', 'Philippines', 'Agriculture', 'Parent', 'Animals'}\n",
      "\n",
      "Loan: 1311731, similarity: 1.0\n",
      "{'Asia', 'Repeat Borrower', 'Woman Owned Biz', 'Philippines', 'Schooling', 'Agriculture', 'Parent', 'Animals'}\n",
      "\n",
      "Loan: 1311963, similarity: 1.0\n",
      "{'Asia', 'Repeat Borrower', 'Woman Owned Biz', 'Elderly', 'Philippines', 'Agriculture', 'Animals'}\n",
      "\n",
      "Loan: 1312459, similarity: 1.0\n",
      "{'Asia', 'Woman Owned Biz', 'Elderly', 'Philippines', 'Agriculture', 'Animals'}\n",
      "\n",
      "Loan: 1309535, similarity: 1.0\n",
      "{'Asia', 'Repeat Borrower', 'Woman Owned Biz', 'Elderly', 'Philippines', 'Agriculture', 'Animals'}\n",
      "\n",
      "\n",
      "***TOP FIVE WORST LOANS for marylynn4377, using dot_product similarity metric***\n",
      "\n",
      "Loan: 1316276, similarity: 0.0\n",
      "{'Services', 'Palestine', 'Arab Youth', 'Unknown', 'Start-Up', 'Youth'}\n",
      "\n",
      "Loan: 1318880, similarity: 0.0\n",
      "{'Retail', 'Africa', 'Mozambique'}\n",
      "\n",
      "Loan: 1307165, similarity: 0.0\n",
      "{'Retail', 'North_America', 'United States'}\n",
      "\n",
      "Loan: 1319446, similarity: 0.0\n",
      "{'Retail', 'Africa', 'Mozambique'}\n",
      "\n",
      "Loan: 1308258, similarity: 0.0\n",
      "{'Rural Exclusion', 'Underfunded Areas', 'Innovative Loans', 'Myanmar (Burma)', 'Unknown', 'Personal Use', 'Youth'}\n"
     ]
    }
   ],
   "source": [
    "print_best_and_worst_loans('marylynn4377', 'dot_product')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Add normalization to introduce diversity\n",
    "Let's introduce some diversity here by taking the square root of the dot product rather than the absolute value itself.  \n",
    "\n",
    "This way, if I take 17 loans in Uganda and 1 in Kenya, the similarity metric will weight Uganda sqrt(17) = 4.1 times as high as Kenya.  Hopefully this will help us from getting loans that are all mostly identical to a person's most common loan profile."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loan profile for brianna9306\n",
      "\n",
      "user_countries\n",
      "\tKenya: 1\n",
      "\tUganda: 17\n",
      "user_tags\n",
      "\tFabrics: 3\n",
      "\tSchooling: 1\n",
      "\tJob Creator: 1\n",
      "\tUnique: 1\n",
      "\tVegan: 2\n",
      "\tRepeat Borrower: 1\n",
      "\tWoman Owned Biz: 9\n",
      "\tAnimals: 1\n",
      "\tSingle Parent: 1\n",
      "\tParent: 9\n",
      "\tEco-friendly: 1\n",
      "\tuser_favorite: 10\n",
      "\tTechnology: 1\n",
      "\tInteresting Photo: 1\n",
      "\tHealth and Sanitation: 1\n",
      "user_sectors\n",
      "\tRetail: 6\n",
      "\tHousing: 1\n",
      "\tAgriculture: 2\n",
      "\tFood: 7\n",
      "\tClothing: 2\n",
      "user_continents\n",
      "\tAfrica: 18\n",
      "user_themes\n",
      "\tSocial Enterprise: 1\n",
      "\tGrowing Businesses: 1\n",
      "\tRural Exclusion: 9\n",
      "\tGreen: 1\n",
      "\tJob Creation: 1\n",
      "\tYouth: 4\n",
      "\tVulnerable Groups: 4\n",
      "\n",
      "\n",
      "***TOP FIVE SIMILAR LOANS for brianna9306, using dot_product similarity metric***\n",
      "\n",
      "Loan: 1310375, similarity: 0.7899117070195137\n",
      "{'Rural Exclusion', 'Africa', 'Repeat Borrower', 'Woman Owned Biz', 'Vegan', 'Mali', 'Parent', 'Food', 'Underfunded Areas', 'user_favorite'}\n",
      "\n",
      "Loan: 1300114, similarity: 0.7732270324689271\n",
      "{'Rural Exclusion', 'Africa', 'Woman Owned Biz', 'Uganda', 'Schooling', 'Parent', 'Food'}\n",
      "\n",
      "Loan: 1313614, similarity: 0.7661479383007253\n",
      "{'Rural Exclusion', 'Africa', 'Repeat Borrower', 'Woman Owned Biz', 'Kenya', 'Agriculture', 'Parent', 'user_favorite', 'Biz Durable Asset', 'Animals'}\n",
      "\n",
      "Loan: 1315583, similarity: 0.7306459217605717\n",
      "{'Africa', 'Repeat Borrower', 'Woman Owned Biz', 'Kenya', 'Retail', 'Schooling', 'Single Parent', 'Parent', 'user_favorite'}\n",
      "\n",
      "Loan: 1310855, similarity: 0.6925475651799606\n",
      "{'Rural Exclusion', 'Africa', 'Woman Owned Biz', 'Kenya', 'volunteer_like', 'Agriculture', 'Parent', 'user_favorite'}\n",
      "\n",
      "\n",
      "***TOP FIVE WORST LOANS for brianna9306, using dot_product similarity metric***\n",
      "\n",
      "Loan: 1310713, similarity: 0.0\n",
      "{'Personal Use', 'Tajikistan', 'Asia'}\n",
      "\n",
      "Loan: 1302510, similarity: 0.0\n",
      "{'Pakistan', 'Transportation', 'Asia', 'Repair Renew Replace', 'Underfunded Areas'}\n",
      "\n",
      "Loan: 1302483, similarity: 0.0\n",
      "{'Pakistan', 'Asia', 'Transportation', 'Elderly', 'Repair Renew Replace', 'Underfunded Areas'}\n",
      "\n",
      "Loan: 1302477, similarity: 0.0\n",
      "{'Pakistan', 'Transportation', 'Asia', 'Repair Renew Replace', 'Underfunded Areas'}\n",
      "\n",
      "Loan: 1315082, similarity: 0.0\n",
      "{'Asia', 'Philippines', 'Transportation'}\n"
     ]
    }
   ],
   "source": [
    "NORMALIZE = 'sqrt'\n",
    "print_best_and_worst_loans('brianna9306', 'dot_product')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This definitely appears to introduce more randomness into the recommendations.\n",
    "\n",
    "It would take A/B tests to validate whether these loans would actually be more persuasive to people and result in more loans given, but at face value it's definitely promising."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}